-
Notifications
You must be signed in to change notification settings - Fork 400
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: add support for microsoft onelake #1564
Conversation
ACTION NEEDED delta-rs follows the Conventional Commits specification for release automation. The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. |
Made code change to test utils to handle abfss (abfss://@onelake.dfs.fabric.microsoft.com) Issue #1418 |
Co-authored-by: Christopher Watford <132389385+watfordkcf@users.noreply.github.com>
Co-authored-by: Christopher Watford <132389385+watfordkcf@users.noreply.github.com>
Co-authored-by: Christopher Watford <132389385+watfordkcf@users.noreply.github.com>
Co-authored-by: Christopher Watford <132389385+watfordkcf@users.noreply.github.com>
@vmuddassir-msft are you getting to a point to where you're ready to put this into Review? |
Delta-rs needs to be upgraded to the latest version of object_store v0.7.0 , in order to add Support for Microsoft Fabric / OneLake . I have upgraded the versions for object_store with other dependencies which reference object_store v0.6.1 I am facing an issue after the upgrade : This is because datafusion seems to be internally using the dependency crate arrow-array v45 while the latest version is arrow-array = { version = "46.0.0" } |
Yes, we need to wait for datafusion to upgrade versions of arrow before we can. Can you upgrade object store independent of arrow? |
Unable to upgrade Object store independently as this is also a dependency for parquet and the latest version of paraquet v46 has linked dependencies to the latest versions of all arrow related crates which includes arrow-array |
@vmuddassir-msft understood. The next version of datafusion is out for vote, so it should be released sometime next week. Once that happens we should be able to get this PR in a merge-able state. :) |
rust/src/writer/record_batch.rs
Outdated
@@ -372,7 +372,8 @@ pub(crate) fn divide_by_partition_values( | |||
}) | |||
.collect::<Result<Vec<_>, DeltaWriterError>>()?; | |||
|
|||
let partition_ranges = lexicographical_partition_ranges(sorted_partition_columns.as_slice())?; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#[allow(warnings)] for now as :lexicographical_partition_ranges is Deprecated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@wjones127 / @rtyler - Datafusion packages have been upgraded
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FYI another dedicated PR was merged to upgrade datafusion, so you'll need to address conflicts with that. Should be quick.
@rtyler - PR is ready to be reviewed The test for onelake is failing as this requires a bearer token , test can be marked as ignored |
@vmuddassir-msft is your expectation that the test will never be run? (in which case we should just delete it), or do you want to configure a bearer token in GitHub secrets so that at least CI can run? |
Head branch was pushed to by a user without write access
Closing this in favor of the rebased #1642 |
No description provided.